Skip to content

fix(train): tch-backend rpath build.rs + unreachable test thresholds + --val-dir / --eval-only (rebased residual) - #1014

Merged
ruvnet merged 1 commit into
ruvnet:mainfrom
stuinfla:fix/trainer-tch024-and-real-validation
Nov 11, 2026
Merged

ruvnet merged 1 commit into
ruvnet:mainfrom
stuinfla:fix/trainer-tch024-and-real-validation

Conversation

@stuinfla

@stuinfla stuinfla commented Jun 11, 2026 •

Copy link
Copy Markdown
Contributor

Rebased onto main: reduced scope

Rebased onto main (dea1cc23, after #2524). Most of the original PR already landed on main through #1008 / ADR-155:

  • the tch 0.24 TryFrom conversions
  • numel cast, EdgeRef import, clamp_min(1.0)
  • floor-division argmax decode
  • the losses.rs E0507 test fix

Those hunks are dropped. This branch is one commit carrying only what main still lacks. It was re-ported onto main's current code, not replayed from the old diff. Authorship is kept (@stuinfla).

What remains, and why it is still needed

  1. build.rs rpath (tch-backend + LIBTORCH_USE_PYTORCH=1).

    • The bug on main: cargo test -p wifi-densepose-train --features tch-backend --lib builds, but the test binary dies at load with libtorch_cpu.so: cannot open shared object file.
    • Scope: now also gated on CARGO_FEATURE_TCH_BACKEND, so it is a strict no-op for default, --no-default-features and wheel builds. It honours PYTHON_SYS_EXECUTABLE the way torch-sys does.
  2. Two gaussian-peak tests asserted unreachable values (still failing on main). A half-pixel keypoint gives a maximum of:

    • exp(−0.5/8) = 0.9394, where the test asserted > 0.95
    • exp(−0.5/4.5) = 0.8948, where the test asserted > 0.9

    The thresholds are now 0.93 / 0.89. The original PR's comment gave 0.8007 and a 0.79 threshold. The measured value is 0.8948, so the comment and the tighter threshold are corrected.

  3. --val-dir <DIR>: a held-out MM-Fi validation set.

    • It takes precedence over main's automatic subject-disjoint split.
    • Any subject overlap with --data-dir is refused as a leak (ADR-155 §Tier-1.2).
    • Main's split and its [SMOKE-TEST] synthetic fallback are unchanged. The original else-branch, which routed synthetic validation through run_training, is not carried, because it would regress ADR-155.
  4. --eval-only --checkpoint <FILE> --val-dir <DIR> [--dump-preds <JSONL>].

    • Trainer::evaluate and the new evaluate_with_dump share one loop (evaluate_inner with an optional sink), so eval-only uses exactly the metric that training validation uses.
    • clap enforces the argument relationships.

Evidence (local; CI does not build tch-backend)

Run with PyTorch 2.11, LIBTORCH_USE_PYTORCH=1 and --test-threads=1.

main dea1cc23 this branch
tch lib tests 353 pass / 2 fail (the two thresholds), plus the loader error unless LD_LIBRARY_PATH is set 355 / 355, no LD_LIBRARY_PATH
cargo test -p wifi-densepose-train --no-default-features — all pass, 0 warnings from touched files
  • Formatting: rustfmt --check is clean on the touched files.
  • CLI end to end: run on a synthetic two-subject MM-Fi fixture (S01 train, S02 held out):
    • --data-dir train --val-dir val trains 3 epochs and logs "REAL held-out validation set … subjects [2]".
    • --eval-only reloads best_*.safetensors, prints EVAL RESULT, and --dump-preds writes 22 JSONL lines (one per window).
    • --val-dir pointing at the training dir is refused, naming the shared subject.
    • clap rejects --eval-only without --checkpoint.
  • Fixture caveat: the fixture is constant data, so its PCK is 0. This shows the plumbing works, not accuracy.
  • Not re-verified: the 40% cross-subject PCK figure in the original description is the contributor's hardware result. It was not reproduced here.

No version bump and no publish. The changes are tch-feature-gated or limited to the train binary, and the published default build is unchanged.

Refs #1010 (closed).

🤖 Generated with claude-flow

@ruvnet

ruvnet commented Jun 17, 2026

Copy link
Copy Markdown
Owner

✅ Reviewed — valuable, and it exposes a CI blind spot. The deeper lesson here is the one worth acting on: the tch-backend feature bit-rotted silently because CI only builds --no-default-features, so the gate never compiled it. Fixing the 13 errors + runtime panics is good; I'd pair it with a CI job that builds --features tch-backend (even build-only) so this can't re-rot. The real --val-dir validation + --eval-only additions are a nice bonus.

…t thresholds, --val-dir and --eval-only (ruvnet#1014)

Rebased residual of ruvnet#1014 onto main. The tch 0.24 compile fixes, clamp_min,
floor-division decode and E0507 test fix from the original PR already landed
on main (ruvnet#1008 / ADR-155), so only what main still lacks is carried here:

- build.rs: with `tch-backend` + LIBTORCH_USE_PYTORCH=1, embed an rpath to
  the PyTorch lib dir. On main the lib unit-test binary dies at load with
  "libtorch_cpu.so: cannot open shared object file". Strict no-op unless the
  feature AND the env var are set (default / --no-default-features / wheel
  builds unchanged). Honours PYTHON_SYS_EXECUTABLE like torch-sys.
- losses.rs: two gaussian-peak tests asserted values that are mathematically
  unreachable for a half-pixel keypoint: exp(-0.5/8) = 0.9394 (< 0.95) and
  exp(-0.5/4.5) = 0.8948 (< 0.9). Thresholds 0.93 / 0.89 with the math.
- trainer.rs: evaluate() and new evaluate_with_dump() share ONE loop
  (evaluate_inner with an optional JSONL sink), so --eval-only measures with
  exactly the metric training validation uses.
- train.rs: --val-dir <DIR> uses a held-out MM-Fi dir as validation, taking
  precedence over the automatic subject-disjoint split; any subject overlap
  with --data-dir is refused as a leak (ADR-155 Tier-1.2). The automatic
  split and the synthetic smoke-test fallback are unchanged.
  --eval-only --checkpoint <FILE> --val-dir <DIR> [--dump-preds <JSONL>]
  loads a checkpoint and runs the validation loop standalone.

Local evidence (PyTorch 2.11, LIBTORCH_USE_PYTORCH=1, --test-threads=1):
main 353 pass / 2 fail (+ loader error without LD_LIBRARY_PATH);
this branch 355/355 with no LD_LIBRARY_PATH.

Co-Authored-By: claude-flow <ruv@ruv.net>
@ruvnet
ruvnet force-pushed the fix/trainer-tch024-and-real-validation branch from 57dc6f7 to 2c03e84 Compare November 11, 2026 13:13
@ruvnet ruvnet changed the title fix(train): un-rot tch-backend vs workspace tch 0.24 + clamp/floor-div runtime bugs + real --val-dir validation and --eval-only mode fix(train): tch-backend rpath build.rs + unreachable test thresholds + --val-dir / --eval-only (rebased residual) Nov 11, 2026
@ruvnet
ruvnet merged commit bf61d4e into ruvnet:main Nov 11, 2026
29 checks passed
#[arg(long, value_name = "FILE", requires = "eval_only")]
dump_preds: Option<PathBuf>,

/// Override the checkpoint output directory from the config.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

log_config_summary(&config);

// ------------------------------------------------------------------
// Eval-only mode: load checkpoint, run the validation loop, exit

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
// Eval-only mode: load checkpoint, run the validation loop, exit
// Eval-only mode: load checkpoint, run the validation loop, exit```suggestion
// Eval-only mode: load checkpoint, run the validation loop, exit**[factory reset]()**

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants